Quantifying the impact of hospital catchment area definitions on hospital admissions forecasts: COVID-19 in England, September 2020-April 2021.

BACKGROUND: Defining healthcare facility catchment areas is a key step in predicting future healthcare demand in epidemic settings. Forecasts of hospitalisations can be informed by leading indicators measured at the community level. However, this relies on the definition of so-called catchment areas or the geographies whose populations make up the patients admitted to a given hospital, which are often not well-defined. Little work has been done to quantify the impact of hospital catchment area definitions on healthcare demand forecasting. METHODS: We made forecasts of local-level hospital admissions using a scaled convolution of local cases (as defined by the hospital catchment area) and delay distribution. Hospital catchment area definitions were derived from either simple heuristics (in which people are admitted to their nearest hospital or any nearby hospital) or historical admissions data (all emergency or elective admissions in 2019, or COVID-19 admissions), plus a marginal baseline definition based on the distribution of all hospital admissions. We evaluated predictive performance using each hospital catchment area definition using the weighted interval score and considered how this changed by the length of the predictive horizon, the date on which the forecast was made, and by location. We also considered the change, if any, in the relative performance of each definition in retrospective vs. real-time settings, or at different spatial scales. RESULTS: The choice of hospital catchment area definition affected the accuracy of hospital admission forecasts. The definition based on COVID-19 admissions data resulted in the most accurate forecasts at both a 7- and 14-day horizon and was one of the top two best-performing definitions across forecast dates and locations. The "nearby" heuristic also performed well, but less consistently than the COVID-19 data definition. The marginal distribution baseline, which did not include any spatial information, was the lowest-ranked definition. The relative performance of the definitions was larger when using case forecasts compared to future observed cases. All results were consistent across spatial scales of the catchment area definitions. CONCLUSIONS: Using catchment area definitions derived from context-specific data can improve local-level hospital admission forecasts. Where context-specific data is not available, using catchment areas defined by carefully chosen heuristics is a sufficiently good substitute. There is clear value in understanding what drives local admissions patterns, and further research is needed to understand the impact of different catchment area definitions on forecast performance where case trends are more heterogeneous.

Characterising information gains and losses when collecting multiple epidemic model outputs.

BACKGROUND: Collaborative comparisons and combinations of epidemic models are used as policy-relevant evidence during epidemic outbreaks. In the process of collecting multiple model projections, such collaborations may gain or lose relevant information. Typically, modellers contribute a probabilistic summary at each time-step. We compared this to directly collecting simulated trajectories. We aimed to explore information on key epidemic quantities; ensemble uncertainty; and performance against data, investigating potential to continuously gain information from a single cross-sectional collection of model results. METHODS: We compared projections from the European COVID-19 Scenario Modelling Hub. Five teams modelled incidence in Belgium, the Netherlands, and Spain. We compared July 2022 projections by incidence, peaks, and cumulative totals. We created a probabilistic ensemble drawn from all trajectories, and compared to ensembles from a median across each model's quantiles, or a linear opinion pool. We measured the predictive accuracy of individual trajectories against observations, using this in a weighted ensemble. We repeated this sequentially against increasing weeks of observed data. We evaluated these ensembles to reflect performance with varying observed data. RESULTS: By collecting modelled trajectories, we showed policy-relevant epidemic characteristics. Trajectories contained a right-skewed distribution well represented by an ensemble of trajectories or a linear opinion pool, but not models' quantile intervals. Ensembles weighted by performance typically retained the range of plausible incidence over time, and in some cases narrowed this by excluding some epidemic shapes. CONCLUSIONS: We observed several information gains from collecting modelled trajectories rather than quantile distributions, including potential for continuously updated information from a single model collection. The value of information gains and losses may vary with each collaborative effort's aims, depending on the needs of projection users. Understanding the differing information potential of methods to collect model projections can support the accuracy, sustainability, and communication of collaborative infectious disease modelling efforts.

Predicting subnational incidence of COVID-19 cases and deaths in EU countries.

BACKGROUND: Recurring COVID-19 waves highlight the need for tools able to quantify transmission risk, and identify geographical areas at risk of outbreaks. Local outbreak risk depends on complex immunity patterns resulting from previous infections, vaccination, waning and immune escape, alongside other factors (population density, social contact patterns). Immunity patterns are spatially and demographically heterogeneous, and are challenging to capture in country-level forecast models. METHODS: We used a spatiotemporal regression model to forecast subnational case and death counts and applied it to three EU countries as test cases: France, Czechia, and Italy. Cases in local regions arise from importations or local transmission. Our model produces age-stratified forecasts given age-stratified data, and links reported case counts to routinely collected covariates (e.g. test number, vaccine coverage). We assessed the predictive performance of our model up to four weeks ahead using proper scoring rules and compared it to the European COVID-19 Forecast Hub ensemble model. Using simulations, we evaluated the impact of variations in transmission on the forecasts. We developed an open-source RShiny App to visualise the forecasts and scenarios. RESULTS: At a national level, the median relative difference between our median weekly case forecasts and the data up to four weeks ahead was 25% (IQR: 12-50%) over the prediction period. The accuracy decreased as the forecast horizon increased (on average 24% increase in the median ranked probability score per added week), while the accuracy of death forecasts was more stable. Beyond two weeks, the model generated a narrow range of likely transmission dynamics. The median national case forecasts showed similar accuracy to forecasts from the European COVID-19 Forecast Hub ensemble model, but the prediction interval was narrower in our model. Generating forecasts under alternative transmission scenarios was therefore key to capturing the range of possible short-term transmission dynamics. DISCUSSION: Our model captures changes in local COVID-19 outbreak dynamics, and enables quantification of short-term transmission risk at a subnational level. The outputs of the model improve our ability to identify areas where outbreaks are most likely, and are available to a wide range of public health professionals through the Shiny App we developed.

Accumulation of Immunity in Heavy-Tailed Sexual Contact Networks Shapes Mpox Outbreak Sizes.

Many countries affected by the global outbreak of mpox in 2022 have observed a decline in cases. Our mathematical model accounting for heavy-tailed sexual partnership distributions suggests that mpox epidemics can hit the infection-derived herd immunity threshold and begin to decline, with <1% of sexually active men who have sex with men infected regardless of interventions or behavioral changes. We consistently found that many countries and US states experienced an epidemic peak, with cumulative cases of around 0.1% to 0.5% among men who have sex with men. The observed decline in cases may not necessarily be attributable to interventions or behavioral changes primarily.

Inferring the proportion of undetected cholera infections from serological and clinical surveillance in an immunologically naive population.

Most infections with pandemic Vibrio cholerae are thought to result in subclinical disease and are not captured by surveillance. Previous estimates of the ratio of infections to clinical cases have varied widely (2 to 100). Understanding cholera epidemiology and immunity relies on the ability to translate between numbers of clinical cases and the underlying number of infections in the population. We estimated the infection incidence during the first months of an outbreak in a cholera-naive population using a Bayesian vibriocidal antibody titer decay model combining measurements from a representative serosurvey and clinical surveillance data. 3,880 suspected cases were reported in Grande Saline, Haiti, between 20 October 2010 and 6 April 2011 (clinical attack rate 18.4%). We found that more than 52.6% (95% Credible Interval (CrI) 49.4-55.7) of the population ≥2 years showed serologic evidence of infection, with a lower infection rate among children aged 2-4 years (35.5%; 95%CrI 24.2-51.6) compared with people ≥5 years (53.1%; 95%CrI 49.4-56.4). This estimated infection rate, nearly three times the clinical attack rate, with underdetection mainly seen in those ≥5 years, has likely impacted subsequent outbreak dynamics. Our findings show how seroincidence estimates improve understanding of links between cholera burden, transmission dynamics and immunity.

Why are different estimates of the effective reproductive number so different? A case study on COVID-19 in Germany.

The effective reproductive number Rt has taken a central role in the scientific, political, and public discussion during the COVID-19 pandemic, with numerous real-time estimates of this quantity routinely published. Disagreement between estimates can be substantial and may lead to confusion among decision-makers and the general public. In this work, we compare different estimates of the national-level effective reproductive number of COVID-19 in Germany in 2020 and 2021. We consider the agreement between estimates from the same method but published at different time points (within-method agreement) as well as retrospective agreement across eight different approaches (between-method agreement). Concerning the former, estimates from some methods are very stable over time and hardly subject to revisions, while others display considerable fluctuations. To evaluate between-method agreement, we reproduce the estimates generated by different groups using a variety of statistical approaches, standardizing analytical choices to assess how they contribute to the observed disagreement. These analytical choices include the data source, data pre-processing, assumed generation time distribution, statistical tuning parameters, and various delay distributions. We find that in practice, these auxiliary choices in the estimation of Rt may affect results at least as strongly as the selection of the statistical approach. They should thus be communicated transparently along with the estimates.

The burden and dynamics of hospital-acquired SARS-CoV-2 in England.

Hospital-based transmission had a dominant role in Middle East respiratory syndrome coronavirus (MERS-CoV) and severe acute respiratory syndrome coronavirus (SARS-CoV) epidemics1,2, but large-scale studies of its role in the SARS-CoV-2 pandemic are lacking. Such transmission risks spreading the virus to the most vulnerable individuals and can have wider-scale impacts through hospital-community interactions. Using data from acute hospitals in England, we quantify within-hospital transmission, evaluate likely pathways of spread and factors associated with heightened transmission risk, and explore the wider dynamical consequences. We estimate that between June 2020 and March 2021 between 95,000 and 167,000 inpatients acquired SARS-CoV-2 in hospitals (1% to 2% of all hospital admissions in this period). Analysis of time series data provided evidence that patients who themselves acquired SARS-CoV-2 infection in hospital were the main sources of transmission to other patients. Increased transmission to inpatients was associated with hospitals having fewer single rooms and lower heated volume per bed. Moreover, we show that reducing hospital transmission could substantially enhance the efficiency of punctuated lockdown measures in suppressing community transmission. These findings reveal the previously unrecognized scale of hospital transmission, have direct implications for targeting of hospital control measures and highlight the need to design hospitals better equipped to limit the transmission of future high-consequence pathogens.

Evaluating the use of social contact data to produce age-specific short-term forecasts of SARS-CoV-2 incidence in England.

Mathematical and statistical models can be used to make predictions of how epidemics may progress in the near future and form a central part of outbreak mitigation and control. Renewal equation based models allow inference of epidemiological parameters from historical data and forecast future epidemic dynamics without requiring complex mechanistic assumptions. However, these models typically ignore interaction between age groups, partly due to challenges in parameterising a time varying interaction matrix. Social contact data collected regularly during the COVID-19 epidemic provide a means to inform interaction between age groups in real-time. We developed an age-specific forecasting framework and applied it to two age-stratified time-series: incidence of SARS-CoV-2 infection, estimated from a national infection and antibody prevalence survey; and, reported cases according to the UK national COVID-19 dashboard. Jointly fitting our model to social contact data from the CoMix study, we inferred a time-varying next generation matrix which we used to project infections and cases in the four weeks following each of 29 forecast dates between October 2020 and November 2021. We evaluated the forecasts using proper scoring rules and compared performance with three other models with alternative data and specifications alongside two naive baseline models. Overall, incorporating age interaction improved forecasts of infections and the CoMix-data-informed model was the best performing model at time horizons between two and four weeks. However, this was not true when forecasting cases. We found that age group interaction was most important for predicting cases in children and older adults. The contact-data-informed models performed best during the winter months of 2020-2021, but performed comparatively poorly in other periods. We highlight challenges regarding the incorporation of contact data in forecasting and offer proposals as to how to extend and adapt our approach, which may lead to more successful forecasts in future.

Scoring epidemiological forecasts on transformed scales.

Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence.

Collaborative nowcasting of COVID-19 hospitalization incidences in Germany.

Real-time surveillance is a crucial element in the response to infectious disease outbreaks. However, the interpretation of incidence data is often hampered by delays occurring at various stages of data gathering and reporting. As a result, recent values are biased downward, which obscures current trends. Statistical nowcasting techniques can be employed to correct these biases, allowing for accurate characterization of recent developments and thus enhancing situational awareness. In this paper, we present a preregistered real-time assessment of eight nowcasting approaches, applied by independent research teams to German 7-day hospitalization incidences during the COVID-19 pandemic. This indicator played an important role in the management of the outbreak in Germany and was linked to levels of non-pharmaceutical interventions via certain thresholds. Due to its definition, in which hospitalization counts are aggregated by the date of case report rather than admission, German hospitalization incidences are particularly affected by delays and can take several weeks or months to fully stabilize. For this study, all methods were applied from 22 November 2021 to 29 April 2022, with probabilistic nowcasts produced each day for the current and 28 preceding days. Nowcasts at the national, state, and age-group levels were collected in the form of quantiles in a public repository and displayed in a dashboard. Moreover, a mean and a median ensemble nowcast were generated. We find that overall, the compared methods were able to remove a large part of the biases introduced by delays. Most participating teams underestimated the importance of very long delays, though, resulting in nowcasts with a slight downward bias. The accompanying prediction intervals were also too narrow for almost all methods. Averaged over all nowcast horizons, the best performance was achieved by a model using case incidences as a covariate and taking into account longer delays than the other approaches. For the most recent days, which are often considered the most relevant in practice, a mean ensemble of the submitted nowcasts performed best. We conclude by providing some lessons learned on the definition of nowcasting targets and practical challenges.

Inferring the differences in incubation-period and generation-interval distributions of the Delta and Omicron variants of SARS-CoV-2.

Estimating the differences in the incubation-period, serial-interval, and generation-interval distributions of SARS-CoV-2 variants is critical to understanding their transmission. However, the impact of epidemic dynamics is often neglected in estimating the timing of infection-for example, when an epidemic is growing exponentially, a cohort of infected individuals who developed symptoms at the same time are more likely to have been infected recently. Here, we reanalyze incubation-period and serial-interval data describing transmissions of the Delta and Omicron variants from the Netherlands at the end of December 2021. Previous analysis of the same dataset reported shorter mean observed incubation period (3.2 d vs. 4.4 d) and serial interval (3.5 d vs. 4.1 d) for the Omicron variant, but the number of infections caused by the Delta variant decreased during this period as the number of Omicron infections increased. When we account for growth-rate differences of two variants during the study period, we estimate similar mean incubation periods (3.8 to 4.5 d) for both variants but a shorter mean generation interval for the Omicron variant (3.0 d; 95% CI: 2.7 to 3.2 d) than for the Delta variant (3.8 d; 95% CI: 3.7 to 4.0 d). The differences in estimated generation intervals may be driven by the "network effect"-higher effective transmissibility of the Omicron variant can cause faster susceptible depletion among contact networks, which in turn prevents late transmission (therefore shortening realized generation intervals). Using up-to-date generation-interval distributions is critical to accurately estimating the reproduction advantage of the Omicron variant.

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States.

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policymakers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision-makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

Human judgement forecasting of COVID-19 in the UK.

Background: In the past, two studies found ensembles of human judgement forecasts of COVID-19 to show predictive performance comparable to ensembles of computational models, at least when predicting case incidences. We present a follow-up to a study conducted in Germany and Poland and investigate a novel joint approach to combine human judgement and epidemiological modelling. Methods: From May 24th to August 16th 2021, we elicited weekly one to four week ahead forecasts of cases and deaths from COVID-19 in the UK from a crowd of human forecasters. A median ensemble of all forecasts was submitted to the European Forecast Hub. Participants could use two distinct interfaces: in one, forecasters submitted a predictive distribution directly, in the other forecasters instead submitted a forecast of the effective reproduction number R t. This was then used to forecast cases and deaths using simulation methods from the EpiNow2 R package. Forecasts were scored using the weighted interval score on the original forecasts, as well as after applying the natural logarithm to both forecasts and observations. Results: The ensemble of human forecasters overall performed comparably to the official European Forecast Hub ensemble on both cases and deaths, although results were sensitive to changes in details of the evaluation. R t forecasts performed comparably to direct forecasts on cases, but worse on deaths. Self-identified "experts" tended to be better calibrated than "non-experts" for cases, but not for deaths. Conclusions: Human judgement forecasts and computational models can produce forecasts of similar quality for infectious disease such as COVID-19. The results of forecast evaluations can change depending on what metrics are chosen and judgement on what does or doesn't constitute a "good" forecast is dependent on the forecast consumer. Combinations of human and computational forecasts hold potential but present real-world challenges that need to be solved.

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021.

BACKGROUND: During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021. METHODS: We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study. RESULTS: We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict. CONCLUSIONS: Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

We compare forecasts of weekly case and death numbers for COVID-19 in Germany and Poland based on 15 different modelling approaches. These cover the period from January to April 2021 and address numbers of cases and deaths one and two weeks into the future, along with the respective uncertainties. We find that combining different forecasts into one forecast can enable better predictions. However, case numbers over longer periods were challenging to predict. Additional data sources, such as information about different versions of the SARS-CoV-2 virus present in the population, might improve forecasts in the future.

Heavy-tailed sexual contact networks and monkeypox epidemiology in the global outbreak, 2022.

The outbreak of monkeypox across non-endemic regions confirmed in May 2022 shows epidemiological features distinct from previously imported outbreaks, most notably its observed growth and predominance amongst men who have sex with men (MSM). We use a transmission model fitted to empirical sexual partnership data to show that the heavy-tailed sexual partnership distribution, in which a handful of individuals have disproportionately many partners, can explain the sustained growth of monkeypox among MSM despite the absence of such patterns previously. We suggest that the basic reproduction number (R0) for monkeypox over the MSM sexual network may be substantially above 1, which poses challenges to outbreak containment. Ensuring support and tailored messaging to facilitate prevention and early detection among MSM with high numbers of partners is warranted.

Comparing human and model-based forecasts of COVID-19 in Germany and Poland.

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.

Simulating respiratory disease transmission within and between classrooms to assess pandemic management strategies at schools.

The global spread of coronavirus disease 2019 (COVID-19) has emphasized the need for evidence-based strategies for the safe operation of schools during pandemics that balance infection risk with the society's responsibility of allowing children to attend school. Due to limited empirical data, existing analyses assessing school-based interventions in pandemic situations often impose strong assumptions, for example, on the relationship between class size and transmission risk, which could bias the estimated effect of interventions, such as split classes and staggered attendance. To fill this gap in school outbreak studies, we parameterized an individual-based model that accounts for heterogeneous contact rates within and between classes and grades to a multischool outbreak data of influenza. We then simulated school outbreaks of respiratory infectious diseases of ongoing threat (i.e., COVID-19) and potential threat (i.e., pandemic influenza) under a variety of interventions (changing class structures, symptom screening, regular testing, cohorting, and responsive class closures). Our results suggest that interventions changing class structures (e.g., reduced class sizes) may not be effective in reducing the risk of major school outbreaks upon introduction of a case and that other precautionary measures (e.g., screening and isolation) need to be employed. Class-level closures in response to detection of a case were also suggested to be effective in reducing the size of an outbreak.

The importance of the generation interval in investigating dynamics and control of new SARS-CoV-2 variants.

Inferring the relative strength (i.e. the ratio of reproduction numbers) and relative speed (i.e. the difference between growth rates) of new SARS-CoV-2 variants is critical to predicting and controlling the course of the current pandemic. Analyses of new variants have primarily focused on characterizing changes in the proportion of new variants, implicitly or explicitly assuming that the relative speed remains fixed over the course of an invasion. We use a generation-interval-based framework to challenge this assumption and illustrate how relative strength and speed change over time under two idealized interventions: a constant-strength intervention like idealized vaccination or social distancing, which reduces transmission rates by a constant proportion, and a constant-speed intervention like idealized contact tracing, which isolates infected individuals at a constant rate. In general, constant-strength interventions change the relative speed of a new variant, while constant-speed interventions change its relative strength. Differences in the generation-interval distributions between variants can exaggerate these changes and modify the effectiveness of interventions. Finally, neglecting differences in generation-interval distributions can bias estimates of relative strength.